GH-38042: [C++][Benchmark] Add non-stream Codec Compression/Decompression#38067
Conversation
|
|
kou
changed the title
| BENCHMARK_TEMPLATE(ReferenceCompression, Compression::LZ4_FRAME); | ||
| BENCHMARK_TEMPLATE(ReferenceStreamingDecompression, Compression::LZ4_FRAME); | ||
| BENCHMARK_TEMPLATE(ReferenceDecompression, Compression::LZ4_FRAME); |
There was a problem hiding this comment.
Is LZ4_FRAME OK?
It seems that Parquet doesn't use LZ4_FRAME.
There was a problem hiding this comment.
We can even benchmark both LZ4 variants.
There was a problem hiding this comment.
It seems that Parquet doesn't use LZ4_FRAME
Aha I remember parquet-mr first implement LZ4. And arrow implement a different version ( LZ4_FRAME ). LZ4 stores an extra-length here.
Maybe apache/parquet-format#168 helps
There was a problem hiding this comment.
And I don't think they have too many differences...
Currently I didn't add LZ4. But feel free to add if neccesssary
github-actions
Bot
added
awaiting merge
|
So we could have added LZ4 and Snappy here. @mapleFU Would you like to do that as a followup PR? |
|
Let me rush it :-) (Just curiously, is it related to #38389 ) ? |
|
It's just reasonable to benchmark all available codecs, not a subset of them. |
|
After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 3be5e60. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them. |
…ompression (apache#38067) ### Rationale for this change Currently, we will enable compression benchmark with ARROW_WITH_BENCHMARKS_REFERENCE Note that it only has benchmark for compressor ( make by Codec::MakeCompressor() ) and decompressor ( make by Codec::MakeDecompressor ). However, Parquet uses Codec to encode and decode. So, I'd like to add benchmarks that use Codec directly. ### What changes are included in this PR? Add benchmark for direct compression and decompression ### Are these changes tested? no need ### Are there any user-facing changes? no * Closes: apache#38042 Authored-by: mwish <maplewish117@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
…ompression (apache#38067) ### Rationale for this change Currently, we will enable compression benchmark with ARROW_WITH_BENCHMARKS_REFERENCE Note that it only has benchmark for compressor ( make by Codec::MakeCompressor() ) and decompressor ( make by Codec::MakeDecompressor ). However, Parquet uses Codec to encode and decode. So, I'd like to add benchmarks that use Codec directly. ### What changes are included in this PR? Add benchmark for direct compression and decompression ### Are these changes tested? no need ### Are there any user-facing changes? no * Closes: apache#38042 Authored-by: mwish <maplewish117@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
3 participants
Rationale for this change
Currently, we will enable compression benchmark with ARROW_WITH_BENCHMARKS_REFERENCE
Note that it only has benchmark for compressor ( make by Codec::MakeCompressor() ) and decompressor ( make by Codec::MakeDecompressor ). However, Parquet uses Codec to encode and decode. So, I'd like to add benchmarks that use Codec directly.
What changes are included in this PR?
Add benchmark for direct compression and decompression
Are these changes tested?
no need
Are there any user-facing changes?
no
CodecCompression/Decompression cases #38042